A Brain-Based Feature Model of Adjective-Noun Composition

Brain-based features of meaning (sensory-motor features: sound, color, manipulation, motion, and shape) are used to compare two popular models of adjective-noun semantic composition: element-wise vector addition and multiplication. A large literature (e.g. Fernandino et al., 2015) suggests that perceptual systems contain information that can be extracted using neural decoding (e.g. Anderson, Murphy & Poesio, 2014). Using Amazon’s Mechanical Turk, participants rated how much each of the words and phrases (made of all combinations of the selected adjectives and nouns) evoked the features. Both multiplication and addition surpass chance at matching the correct phrase, but addition outperformed multiplication (addition = 7.6/60, multiplication = 13.4/60). Addition allows the adjective to weight the important sensory-motor attributes for the noun. Based on these behavioral results, we predict, and will test in upcoming work, that addition will also be successful when using brain activity (from fMRI) as the representations of the adjectives, nouns, and phrases.